The disclosure is generally directed to archiving of data, in particular point-in-time copies referred to as “snapshots” or “snaps”, to cloud-based storage.
“Cloud-based” storage refers to data storage as a service offered to clients or customers, having a generalized object-based interface and generally wide-area availability (i.e., ability to connect via the public internet). In particular, the disclosure relates to archiving user-created snaps for all type of objects supported in a data storage system, such as logical units (LUN), file systems (FS), datastores, virtual volumes (vVOLs), etc. A gateway device is used to interface one or multiple data storage systems to a cloud service. In one example, access is provided using the S3 API to a cloud solution such as Azure, Virtustream, Amazon, etc. A storage administrator may be able to define whether the solution is optimized for backup or restore, or strike a compromise between the two by adjusting the frequency of establishing a common base and whether the backups should be incremental, differential or a mix of both.
More particularly, cloud service providers (CSPs) like Amazon offer cloud storage services wherein data can be stored as a set of ‘objects’ in a ‘bucket’. There can be one or more buckets associated with a customer ‘account’. A CSP may offer an interface such as a RESTful API (REST API) for managing storage. The REST API uses HTTP primitives such as PUT, GET, and DELETE to save and retrieve data. A CSP may charge its customers on two fronts:                Amount of storage used in the cloud        Number of REST API calls made        
An object is a fixed size contiguous series of data with a maximum size, e.g., 5 TB. An object is treated as a whole and is saved and retrieved as one unit. Although an object can be uploaded in multiple parts (e.g., up to 10,000), it cannot be partially updated or read. For example, if an object of size 1 MB has been uploaded to the cloud, the client cannot update just a 50 KB range of data somewhere within the 1 MB extent of the object, such as at the beginning or end for example. The entire object must be written or read. This limitation offers challenges in how cloud storage can be used for periodic archival of snapshots.